The Varnish Configuration Language (VCL), I'm sure you know already, is the source of Varnish versatility: by only enforcing the protocol flow and leaving the business logic to the user, Varnish can be easily configured to do things far beyond caching.

However, because the logic of websites is generally focused around hosts, and the VCL thinks in terms of processing steps, configuration may sometimes a bit odd, with the need to place safeguards around your code to ensure that logic for one host isn't applied to another one.

It works, but it can be tedious and unwieldy, so today we are going to have a look at how we can silo our VCL per website to achieve better maintainability.

What's the issue, really?

Let's start with an example: imagine you want to serve two domains:"api.example.com" and "cutecatz.example.com" (I know, I always have very bad examples; don't judge me!). The latter features highly cacheable kitten pictures, while the former is protected by a token and its content can only be cached for a few seconds.

Your VCL could look like this:

vcl 4.0;

backend be_api {
	.host = "192.168.0.10";
}

backend be_catz {
	.host = "192.168.0.20";
}

sub vcl_recv {
	if (req.http.host == "api.example.com") {
		set req.backend_hint = be_api;
		if (req.http.token == "ItSOKYouCanGoThrough") {
			return (hash);
		} else {
			return (synth(403));
		}
	} else if (req.http.host == "cutecatz.example.com") {
		set req.backend_hint = be_catz;
		return (hash);
	} else {
		return (synth(404));
	}	
}

sub vcl_backend_response {
	if (bereq.http.host == "api.example.com") {
		set beresp.ttl = 2s;
	} else if (bereq.http.host == "cutecatz.example.com") {
		set beresp.ttl = 1w;
	}
}

It's nice and all, but you basically need to enclose each logical block inside a Host test, which isn't very sexy when you have two hosts, and cumbersome when you have hundreds of domains (multi-tenancy, anyone?). Instead, it would be better to be able to write two VCL files, one for each site:

api.vcl:

vcl 4.0;

backend be_api {
	.host = "192.168.0.10";
}

sub vcl_recv {
	set req.backend_hint = be_api;
	if (req.http.token == "ItSOKYouCanGoThrough") {
		return (hash);
	} else {
		return (synth(403));
	}	
}

sub vcl_backend_response {
	set beresp.ttl = 2s;
}

catz.vcl:

vcl 4.0;

backend be_catz {
	.host = "192.168.0.20";
}

sub vcl_recv {
	set req.backend_hint = be_catz;
	return (hash);
}

sub vcl_backend_response {
	set beresp.ttl = 1w;
}

Enter labels and the cli file

It turns out that we have been able to do this since Varnish 5.0, but that feature sort of flew under the radar. Fortunately, it's quite easy to enable!

The first thing to do is to create a root (or "route") VCL that will select the right VCL to use for a given host:

vcl 4.0;

# fake, never-used backend
# to silence the compiler
backend fake {
	.host = "0:0";
}
sub vcl_recv {
	if (req.http.host == "api.example.com") {
		return (vcl(label-api));
	} else if (req.http.host == "cutecatz.example.com") {
		return (vcl(label-catz));
	} else {
		return (synth(404));
	}	
}

The only tricky part is the "return (vcl($LABEL));" part, mostly because I haven't explained what a label is, silly me. Simply put, it's a symbolic name hiding a "true" VCL and with it, we'll be able to change the underlying VCL without changing the root VCL. What we are preparing here is a tree of vcl/labels that we need to load into Varnish:

It's easy enough, but there's one thing that will require our attention though: we can't have dangling routes or label (the destinations MUST exist) so we have to load the elements from bottom to top:

# load both "real" VCLs
varnishadm vcl.load vcl-api-orig /etc/varnish/api.vcl
varnishadm vcl.load vcl-catz-orig /etc/varnish/catz.vcl
# apply labels
varnishadm vcl.label label-api vcl-api-orig
varnishadm vcl.label label-api vcl-api-orig
# load root.vcl, and make it the entry point
varnishadm vcl.load vcl-root-orig /etc/varnish/root.vcl
vcl.use vcl-root-orig

See? Easy-peasy! But we want to do this every time we start, so Varnish had a switch, named "-I" that'll parse the commands to feed varnishadm. We can then start varnishd using:

varnishd -I start.cli -F -a :80 -f ''

with start.cli containing:

vcl.load vcl-api-orig /etc/varnish/api.vcl
vcl.load vcl-catz-orig /etc/varnish/catz.vcl
vcl.label label-api vcl-api-orig
vcl.label label-catz vcl-catz-orig
vcl.load vcl-root-orig /etc/varnish/root.vcl
vcl.use vcl-root-orig

As you can see, it's really just a matter of placing the varnishadm commands inside a file, as they would be typed directly inside the varnishadm console. Note, we can use the "vcl.list" command to get glance of what is loaded and how things are linked.

# varnishadm vcl.list
active      auto/warm          - vcl-api-orig (1 label)
available   auto/warm          - vcl-catz-orig (1 label)
available  label/warm          - label-api -> vcl-api-orig (1 return(vcl))
available  label/warm          - label-catz -> vcl-catz-orig (1 return(vcl))
available   auto/warm          - vcl-root-orig

What about reloading?

We could restart varnishd every time for either api.vcl or catz.vcl, but that wouldn't be very nice, would it? And thanks to labels, we don't have to.

The first thing is to load our new VCL:

varnishadm vcl.load vcl-api-$(date +%s) /tmp/api.vcl

and maybe check that it got loaded:

#: varnishadm vcl.list
active      auto/warm          - vcl-api-orig (1 label)
available   auto/warm          - vcl-catz-orig (1 label)
available  label/warm          - label-api -> vcl-api-orig (1 return(vcl))
available  label/warm          - label-catz -> vcl-catz-orig (1 return(vcl))
available   auto/warm          - vcl-root-orig
available   auto/warm          - vcl-api-1523889985

Great, it's here! (not really a surprise, but still, nice). Note one very nice thing along the way: even though we loaded the same file, internally, it's a new VCL object. That means that if needed, we can rollback to the previous version! (That is why I timestamped it.)

Now we just need to change the label, which is done exactly the same way as when we first created it:

varnishadm vcl.label label-api vcl-api-1523889985

And...we're done? Yes we are, the new VCL is now used by new request. If you want to be extra tidy, you can remove the previous VCL from the list:

varnishadm vcl.discard vcl-api-orig

However, make sure you don't need to rollback! Seriously, VCLs are quite tiny, and leaving them around isn't a big issue for the system, so keep a few if you can.

Adding a new domain

There's one thing left to do: look at what adding a new domain means in terms of configuration file as well as upgrade commands.

The first one is easy, if we want to add cutedogz.example.com to the mix, we need to update both root.vcl:

vcl 4.0;

# fake, never-used backend
# to silence the compiler
backend fake {
	.host = "0:0";
}
sub vcl_recv {
	if (req.http.host == "api.example.com") {
		return (vcl(label-api));
	} else if (req.http.host == "cutecatz.example.com") {
		return (vcl(label-catz));
 +   } else if (req.http.host == "cutedogz.example.com") {
 +       return (vcl(label-dogz));
	} else {
		return (synth(404));
	}	
}

and start.cli:

vcl.load vcl-api-orig /etc/varnish/api.vcl
vcl.load vcl-catz-orig /etc/varnish/catz.vcl
+ vcl.load vcl-dogz-orig /etc/varnish/dogz.vcl
vcl.label label-api vcl-api-orig
vcl.label label-catz vcl-catz-orig
+ vcl.label label-dogz vcl-dogz-orig
vcl.load vcl-root-orig /etc/varnish/root.vcl
vcl.use vcl-root-orig

Next time varnishd starts, the new host will be handled. But we don't want to wait until then, we want cute puppies NOW!

To do so, we will have to:

• load cutedogz.vcl
• give it a label
• load the new root VCL
• make it active

This translates pretty naturally into:

varnishadm vcl.load vcl-dogz-orig /etc/varnish/dogz.vcl
varnishadm vcl.label label-dogz vcl-dogz-orig
varnishadm vcl.load vcl-root-new /etc/varnish/root.vcl
varnishadm vcl.use vcl-root-new

And that's it!

The sky is the limit

This was quite a quick post, but as you can see, the logic is pretty simple as long as you remember a few basic points:

• you must load a VCL before labelling it
• you must return labels, not VCLs
• you can only branch to a label from vcl_return, and only once
• you can't load a VCL returning a non-existent label

Once you know these, you are free to explore! I chose the example of virtual host here because it's natural and many users have that use case, but the same can be done for listening ports, or sub-directory, or even user roles. Really, these mechanics start being useful as soon as you can make a routing decision based on the request.

One thing left as an exercise to the reader is how to use systemd's %i to reload a specific host without clunky scripts. That's a topic for a future post, but if you have questions, you know where to find me ;-)

Dig into more Varnish and VCL tips and tricks.